Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Multi gpu tests and predict_tile error #646 #649

Closed
wants to merge 2 commits into from
Closed

[WIP] Multi gpu tests and predict_tile error #646 #649

wants to merge 2 commits into from

Conversation

bw4sz
Copy link
Collaborator

@bw4sz bw4sz commented Apr 1, 2024 •
edited
Loading

This is still a work in progress, closes #646 and #467.

  • create a test that reproduces and can bypass if not multi-gpu
  • identify source of error.

@bw4sz bw4sz changed the title Multi gpu tests and predict_tile error #646 [WIP] Multi gpu tests and predict_tile error #646 Apr 1, 2024
@bw4sz
Copy link
Collaborator Author

bw4sz commented Apr 3, 2024

Profile results of moving cpu() call to the end of the prediction epoch.

# Profile predict_tile
from deepforest import main
import time

# Profile predict_tile
path = "/orange/ewhite/NeonData/OSBS/DP3.30010.001/neon-aop-products/2023/FullSite/D03/2023_OSBS_7/L3/Camera/Mosaic/2023_OSBS_7_406000_3291000_image.tif"
m = main.deepforest()
m.use_release()

start_time = time.time()
m.predict_tile(path)
end_time = time.time()

execution_time = end_time - start_time
print(f"Execution time: {execution_time} seconds")

On HPC with 2 available GPUs.
Original: (fails due to multi-gpu)

PR: 43.12

@bw4sz bw4sz closed this Apr 3, 2024
@bw4sz bw4sz deleted the multi_gpu_tests branch April 3, 2024 18:43
@bw4sz bw4sz mentioned this pull request Apr 3, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

predict_tile() output is incorrect when multiple GPUs are present
1 participant
@bw4sz